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Biomedical  Application  of  Target  Tracking  in  Clutter 
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Virginia  Image  and  Video  Analysis  (VIVA) 

Department  of  Electrical  Engineering 
University  of  Virginia 
Charlottesville,  VA  22904  USA 


Abstract 

The  movement  of  leukocytes  (white  blood  cells)  and 
their  interaction  with  the  endothelium  (vessel  wall) 
provides  valuable  information  about  the  mechanism  of 
inflammation  and  inflammatory  disease.  In  order  to 
investigate  leukocyte  motion  within  living  animals , 
advanced  automated  tracking  algorithms  are  requisite. 
We  introduce  military  target  tracking  algorithms  for 
the  purpose  of  tracking  cell  movement.  In  33 
experiments,  we  compare  the  tracking  performance  of 
five  trackers.  The  trackers  tested  include  the  centroid 
trackery  the  correlation  tracker ,  an  enhanced  centroid 
tracker,  an  enhanced  correlation  tracker  and  an  active 
contour  (snake)  tracker.  Of  the  five  methods,  the  snake 
tracker  proved  to  be  the  most  robust  method  in  terms  of 
the  highest  percentage  of  frames  tracked  and  the 
lowest  root  mean-squared  error.  The  paper  provides 
an  overview  of  the  five  trackers  and  gives  experimental 
results. 

1.  Introduction 

A  rolling  leukocyte  is  an  activated  white  blood  cell 
that  travels  along  the  endothelium  wall  to  the  site  of 
injury.  Tracking  rolling  leukocytes  during 
inflammation  is  critical  for  analysis  of  the 
inflammatory  process.  Previous  methods  used  to  track 
cell  movements  in  vitro  (in  flow  chambers)  with 
centroid  and  correlation  tracking  algorithms  were 
moderately  successful,  but  proved  insufficient  in  vivo 
(in  living  animals).  Although  intravital  tracking  is  more 
difficult,  it  provides  valuable  characterization  of  the 
cell-vessel  interaction  that  is  not  possible  within  a  flow 
chamber.  This  paper  compares  the  performance  of  the 
basic  centroid  and  correlation  trackers,  enhanced 
centroid  and  correlation  trackers,  and  an  active 
contour-based  snake  tracker.  This  comparison 
evaluates  the  algorithm  robustness  for  tracking  cells 
within  living  animals  using  video  microscopy. 

Active  contour-based  tracking  shows  significant 
potential  for  in  vivo  applications.  Since  we  know  the 


general  shape  and  scale  of  the  cells  to  be  tracked,  we 
can  constrain  active  contours  to  capture  such  shapes. 
Our  initial  study  concludes  that  the  snake  tracker 
algorithm  results  are  superior  to  the  basic  and  enhanced 
centroid  and  correlation  trackers.  These  snake 
algorithms  will  be  further  integrated  into  a  robust 
automatic  in  vivo  cell  tracking  system.  Such  a  system 
would  eliminate  the  current  time-consuming  technique 
of  manual  tracking  to  observe  rolling  leukocytes. 
Furthermore,  automated  tracking  will  remove 
investigator  bias  from  tracking  data. 

2.  Methods 

We  evaluated  five  trackers  that  represent  a  broad 
spectrum  of  tracking  methodologies.  The  conventional 
centroid  and  correlation  trackers  are  included  in  the 
study.  Enhanced  versions  of  the  centroid  and 
correlation  trackers  are  also  analyzed.  Finally,  we 
scrutinize  a  tracker  based  on  active  contours.  Here,  we 
briefly  review  the  approach  taken  by  each  tracking 
system. 

2.1  The  Centroid  Tracker 

The  first  tracking  algorithm  implemented  is  a  simple 
centroid  tracker.  The  fundamental  idea  of  the  centroid 
tracker  is  to  compute  the  "center  of  mass"  within  a 
subimage  to  locate  the  center  of  the  target.  Here, 
"mass"  is  the  sum  of  intensities  in  the  subimage  called 
the  track  gate.  We  automatically  compute  the  track 
gate  boundary  to  bound  the  target  cell.  Within  this 
track  gate,  we  use  the  pixel  intensities  to  calculate  the 
center  of  mass.  If  sufficient  contrast  exists  between  the 
cell  and  the  background,  the  centroid  tracker  gives 
reliable  results.  Where  contrast  varies  or  the  scene  is 
cluttered,  the  centroid  is  apt  to  lose  track  of  the  target 
cell.  In  the  cluttered  environment  of  intravital  cell 
tracking,  the  centroid  tracker  is  prone  to  locking  on  to 
brighter  cells  or  to  the  endothelium  (vessel  wall). 


This  material  is  based  upon  work  supported  in  part  by  the 
U.  S.  Army  Research  Laboratory  and  the  U.  S.  Army  Research 
Office  under  contract/ grant  number  DAAD19-01-1-0594 


2.2  The  Correlation  Tracker 


The  second  tracking  algorithm  implemented  is  a 
basic  correlation  tracker.  The  fundamental  idea  of  a 
correlation  tracker  is  to  locate  the  cell  though  template 
matching.  Specifically,  the  correlation  tracker  creates  a 
template  of  the  leukocyte  to  be  tracked  from  the  initial 
track  gate,  then  uses  this  template  to  find  the  highest 
normalized  cross  correlation  measure  within  future 
frames  to  locate  the  cell  position.  For  template  T  and 
subimage  5,  the  normalized  cross  correlation  at 
translation  (a,  b)  is  given  by 
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The  correlation  tracker  is  straightforward  to 
implement,  and  is  able  to  detect  objects  that  do  not 
change  size,  shape,  shade  or  move  great  distances  from 
the  previous  frame.  Because  leukocytes  in  vitro  remain 
relatively  stable  visually,  the  correlation  is  fairly 
successfiil  in  a  flow  chamber  environment.  However, 
the  cluttered  videos  obtained  in  vivo  are  beyond  the 
capability  of  this  basic  tracker. 


2.3  The  SuperCentroid  Tracker 

The  third  tracking  algorithm  tested  is  an  enhanced 
centroid  tracker  called  the  SuperCentroid.  The 
fundamental  difference  between  SuperCentroid  and  the 
conventional  centroid  tracker  is  that  SuperCentroid 
uses  adaptive  template  matching  to  determine  when  the 
target  object  is  lost.  Adaptive  template  matching  uses 
an  adaptive  target  template  to  locate  the  object  of 
interest  by  matching  the  template  to  the  object  in  the 
image  and  allowing  the  most  recent  target  image  to 
gradually  alter  the  template.  Adaptive  template 
matching  gives  the  ability  to  track  cells  that  change 
size,  shape,  and  intensity  due  to  cell  deformation  or 
camera  focus,  angle,  and  lighting.  The  template  T*+i  is 
updated  by  taking  a  weighted  average  of  the  current 
track  gate  O*  with  the  current  template  T*: 

T*+1  =  }T*  +  (l-y)0*  (2) 

where  both  T*  and  O*  are  centered  at  the  centroid. 


2.4  The  SuperCorrelation  Tracker 

The  fourth  tracking  algorithm  in  the  study  is  an 
enhanced  correlation  tracker  called  the 
SuperCorrelation  tracker.  SuperCorrelation,  like  the 
SuperCentroid,  uses  adaptive  template  matching. 
SuperCorrelation  also  uses  the  centroid  to  re-position 
the  center  of  the  template.  In  addition  to  the 
mechanism  of  adaptive  template  matching,  both  the 


SuperCentroid  and  the  SuperCorrelation  trackers  utilize 
several  additional  enhancements. 

2.5  Common  Features  of  Enhanced  Trackers 

Both  SuperCentroid  and  SuperCorrelation  increase 
tracker  robustness  by  implementing  background 
removal,  image  enhancement  and  the  Kalman  filter. 
Background  removal  extracts  non-moving  features 
such  as  muscle,  tissue,  and  other  vessel  walls.  It  is 
accomplished  by  time-averaging  the  video  frames  in  a 
sequence  and  subtracting  the  averaged  frame  from  each 
video  frame  while  tracking. 

Given  that  the  cells  are  roughly  circular  in  shape  and 
have  a  known  diameter,  we  can  utilize  morphological 
filters  to  further  reduce  clutter.  We  have  used  both 
morphological  anisotropic  diffusion  [1]  and  standard 
morphological  filters  for  enhancement.  In  this  study, 
we  report  results  with  the  standard  morphological 
open-close  filter: 

E  =  (I  o  B)  •  B  (3) 

where  E  is  the  enhanced  image,  I  is  the  input  image, 
and  B  is  a  structuring  element  that  is  circular  and  set  to 
be  of  smaller  scale  than  the  cells. 

The  third  enhancement  common  to  both  the 
SuperCentroid  and  SuperCorrelation  trackers  is  the  use 
of  a  predictive  filter  for  coasting.  The  Kalman  filter 
provides  a  means  to  predict  leukocyte  movement  when 
the  leukocyte  is  obscured  by  clutter.  Essentially,  the 
Kalman  filter  uses  the  history  of  the  leukocyte  position 
to  the  position  in  the  subsequent  frame.  If  the  tracker 
loses  the  target  (as  detected  by  poor  correlation),  the 
Kalman  filter  prediction  can  be  used  as  the  observation. 
In  this  case,  we  say  that  the  tracker  is  coasting . 

We  form  two  Kalman  filters  for  prediction:  one  for 
the  horizontal  position  of  the  leukocyte  and  one  for  the 
vertical  position  of  the  leukocyte  [2].  Let  ik  be  the  row 
position  of  a  cell  at  video  frame  k .  Then 

ik+i  =  4+  St  Vlk  (4) 

where  St  is  the  time  between  frames  (here  1/30  s)  and 
V ^  is  the  velocity.  The  predicted  row  position  is 

=  lk\k  +  St  VlJc+l\k  (5) 

A 

where  is  the  filtered  row  position  estimate,  and 

Vk+l\k  *s  Pre<^ctec*  velocity.  Once  we  have 

tracked  a  cell  for  a  few  frames,  the  Kalman  gains  will 
give  more  weight  to  the  predicted  position  and  velocity 
over  the  observed  position  and  velocity.  The  Kalman 
filter  provides  the  minimum  mean-squared  error 
solution  to  the  track  position  estimation  and  prediction 


problem  [2],  given  the  assumption  of  a  constant 
velocity  target. 

2.6  The  Snake  Tracker 


The  fifth  tracking  algorithm  we  implemented  is  an 
active  contour  or  snake  tracker.  The  snake  algorithm 
tries  to  minimize  an  energy  function  based  on  internal 
snake  energy  (tension  and  rigidity),  external  energy 
(co-location  with  maxima  in  image  gradient),  shape, 
size,  position,  and  sampling  constraints.  These 
constraints  are  weighted  and  combined  into  an  energy 
functional  [3]: 


^snake  —^l^int  +  ^2^ext  ^3^shape 

(6) 

+  A4  Es[Zq  +  ^5^pos  +  ^6  ^sampling  • 


The  internal  energy  is  the  bending/stretching  energy 
that  is  minimized  by  low  magnitudes  of  the  first  and 
second  derivatives  of  the  curve  C(s)  with  respect  to  the 
parameter  s .  The  external  energy  is  minimized  when 
the  contour  is  located  over  the  edge  of  a  cell  -  the 
position  of  high  gradient  magnitude. 

These  snakes  are  well-suited  for  contrast-changing 
and  shape-changing  cells.  Because  we  know  shape  size 
and  position  information  about  cells  a  priori  we  can 
incorporate  these  as  parameters  in  our  energy  function 
to  provide  a  more  robust  algorithm.  Snake  shape  can  be 
constrained  to  known  elliptical  and  circular  cell  shapes. 
Snake  size  is  known,  because  we  know  the 
approximate  cell  size  from  the  scale  of  the  video. 
Figure  1  shows  the  importance  of  the  combination  of 
the  shape  and  size  constraints  in  capturing  a  leukocyte 
for  tracking. 

The  position  constraint  incorporates  future  cell 
positions  as  predicted  by  a  Kalman  filter.  The  snake 
used  here  is  parametric,  described  by  several  (-100) 
discrete  positions  known  as  snaxels ,  which  outline  the 
target  object.  As  the  snake  evolves  with  cell  variations, 
the  snaxel  positions  adjust  to  keep  an  even  spacing 
around  the  target  object  boundary.  In  addition,  the 
snake  re-parameterizes  itself  (keeping  uniform 
sampling)  according  to  the  sampling  energy  term. 
Euler  equations  derived  from  (6)  provide  updates  for 
the  (jc,  y)  position  of  each  snaxel  [3]. 

The  assumed  initial  radius  of  the  snake  tracker  is 
integral  to  effective  tracking.  This  parameter  allows  the 
snake  to  formulate  the  size  constraint,  which  governs 
the  relative  size  of  the  circle  or  ellipse  shape  constraint. 
An  accurate  value  of  the  initial  radius  as  compared  to 
the  actual  radius  of  the  target  cell  realizes  a  more 
effective  snake  tracker. 


Figure  1:  (a)  Leukocytes  within  a  video  frame,  (b) 
Tracking  with  shape  constraint  only,  (c)  Tracking  with 
size  constraint  only,  (d)  Tracking  with  both  shape  and 
size  constraints.  The  initial  and  the  final  snakes  are 
shown  in  white  and  black  respectively  [3]. 

2.7  Tracker  Performance  Measures 

We  determine  a  tracker’s  performance  by  the 
percentage  of  frames  tracked  and  the  root  mean- 
squared  error  (RMSE).  The  percentage  of  frames 
tracked  is  the  number  of  frames  tracked  by  the  tracker 
divided  by  the  number  of  frames  the  cell  is  in  the  video 
sequence.  Here,  we  count  a  frame  as  "tracked"  if  the 
computed  position  is  within  one  cell  radius  of  the 
manually  determined  position.  The  RMSE  (in  microns) 
describes  how  accurately  the  tracker  tracks  the  cell  as 
compared  to  the  “ground  truth”  data. 

The  combination  of  the  percentage  of  frames  tracked 
and  the  RMSE  yields  the  qualitative  performance 
ratings  shown  in  Table  1. 


Table  1:  Tracker  performance  ratings 


Performance 

Rating 

%  Frames 
Tracked 

RMSE 

(microns) 

Excellent 

90-100 

below  1 

Good 

75-89 

1-2 

Fair 

60-74 

3-5 

Poor 

below  60 

above  5 

3.  Results  and  Conclusions 

Our  dataset  consists  of  intravital  microscopy  video 
recordings  of  rolling  leukocytes  in  the  mouse  cremaster 
muscle  observed  via  transillumination.  We  tracked 
thirty-three  cells  from  ten  venules  (small  vessels).  Five 
sets  are  TNF-a  treated  venules;  five  sets  are  untreated. 
The  TNF-a  treatment  increases  the  inflammatory 
response  and  thus  slows  down  the  rolling  cells.  Such 
cells  are  then  more  easily  tracked  at  the  lower 


velocities.  Each  video  sequence  contains  the  tracked 
cells  for  about  2  seconds  (or  for  60  frames  at  30 
frames/second).  The  efficacy  of  each  tracker  is 
determined  by  comparing  experimental  results  with 
“ground  truth”  data  -  a  manual  frame-by-frame  record 
of  cell  position.  Currently  all  five  trackers  acquire  the 
initial  position  from  the  first  “ground  truth”  position 
value.  We  intend  to  integrate  automatic  cell 
identification  in  future  tracking  systems. 

The  snake  tracker  produced  results  with  the  smallest 
average  RMSE  and  largest  average  percentage  of 
frames  tracked.  The  results  attest  to  the  snake  tracker's 
robust  ability  to  track  cells  in  cluttered  conditions.  The 
snake  tracker  is  able  to  overcome  obstacles  that  inhibit 
the  other  trackers  such  as  bright  cells  in  the 
neighborhood  of  the  target  cell,  low  contrast  with  the 
background,  and  fast  moving  cells. 

The  numerical  results  yielded  by  the  snake  tracker 
prove  the  potential  for  a  robust  in  vivo  cell  tracker.  As 
shown  in  Figure  2,  the  snake  tracker  is  able  to  track 
100%  of  the  frames  in  all  16  TNF-a  data  sets.  In 
addition,  the  average  RMSE  for  the  16  TNF-a 
sequences  is  0.33  microns,  for  cells  that  measure  ~10 
microns  in  diameter.  The  performance  of  the 
SuperCorrelation  tracker  was  acceptable,  but  the 
average  RMSE  of  1.94  is  over  five  times  that  of  the 
snake  tracker.  The  SuperCentroid,  centroid,  and 
correlation  trackers  do  not  perform  nearly  as  well  as 
evidenced  by  Figure  2.  In  the  untreated  venules,  the 
performance  of  the  snake  tracker  drops  to  48%  of  the 
frames  tracked  successfully.  Nevertheless,  the 
performance  of  the  other  four  trackers  also  declines, 
with  the  SuperCentroid  providing  the  next  highest 
tracking  percentage  at  35%.  In  this  case,  the  snake 
tracker  also  has  the  lowest  RMSE  by  a  margin  of  7 
microns. 


1  2 _ 3 _ 4 _ 5 


Figure  2:  Comparison  of  average  RMSE  (in 
microns)  and  percent  tracked  for  the  experiments  using 
treated  venules. 

Figure  3  displays  the  hand-tracked  displacements  of 
the  target  cell  from  the  initial  position  (ground-truth), 


and  each  tracker’s  attempt  to  follow  the  cell  for  a  TNF- 
a  sequence.  Clearly,  the  snake  tracks  the  target  cell 
more  accurately  than  the  other  algorithms  in  this  case. 
Similar  results  are  observed  from  the  other  32  data  sets. 

The  snake  tracker  can  be  improved  in  the  following 
areas:  computational  complexity,  initial  radius 
specification,  and  clutter-reducing  methods.  Currently, 
the  snake  tracker  cannot  produce  results  in  real-time. 
For  example,  execution  of  the  snake  tracker  on  a 
ninety-frame  sequence  using  a  desktop  machine  with  a 
Pentium  3  processor  and  256MB  of  RAM  takes  16 
minutes  to  complete.  Thus,  the  snake  tracker  needs 
about  13  hours  to  process  50  ninety-frame  sequences. 
Eventually,  we  hope  to  design  and  implement  a  system 
that  can  process  sequences  at  or  near  real  time.  To 
reduce  computation  time  and  attain  this  goal,  we  are 
investigating  both  hardware  and  algorithmic 
enhancements.  The  specific  algorithmic  enhancements 
include  fast  matrix  inversion  for  calculation  of  the 
snake  update  equations. 

Furthermore,  the  snake  tracker  will  be  made  more 
robust  once  we  can  effectively  and  efficiently  identify 
the  initial  radius  of  the  cell.  Presently,  we  use  a  range 
of  radius  values  to  achieve  tracking  for  a  given  cell. 
More  research  is  needed  to  develop  a  means  to 
automatically  provide  the  snake  tracker  with  an  initial 
cell  radius.  The  snake  tracker  can  be  further  improved 
by  incorporating  additional  filters  that  remove  clutter 
while  maintaining  target  edges  at  a  range  of  scales. 

A  user-friendly  software  package  for  laboratory 
application  is  another  step  in  realizing  a  complete  in 
vivo  cell  tracking  system.  We  have  prototyped  the 
Python ,  a  graphical  user  interface  that  allows  user- 
friendly  cell  selection  and  automated  tracking.  The 
Python  provides  for  manual  target  acquisition  and 
specification  of  various  tracking  algorithms  and  clutter- 
reducing  filters.  A  sample  screen  view  of  the  Python  is 
shown  in  Figure  4. 

Snake-based  algorithms  show  significant  promise  for 
future  in  vivo  cell  tracking.  The  ability  to  track  rolling 
leukocytes  in  vivo  will  bolster  research  and 
development  in  anti-inflammatory  drugs  for  diseases 
such  as  Crohn’s  disease  and  atherosclerosis. 
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Figure  3:  Comparison  of  tracker  displacements  for  a  sequence  obtained  from  a  TNF-  a  treated  vessel. 


Figure  4:  Sample  screen  view  of  Python  graphical  user  interface  acquiring  three  cells. 
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